Repository navigation
test(continuity): add suites for gateway restart and package upgrade - #3521
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-3521.docs.buildwithfern.com/openshell |
|
/ok-to-test 878d186 |
|
/ok-to-test c8ab35d |
3ca9110 to
c9f262b
Compare
|
/ok-to-test c9f262b |
|
The upgrade qualification is currently blocked by a product-level persisted-storage compatibility issue between Both the DEB and RPM scenarios now pass guest readiness, install the retained prerelease, create the source sandbox, and install the candidate packages. After the candidate gateway restarts, it cannot decode the persisted sandbox protobuf. The observed failures report wire-type mismatches for Initial tracing points to the |
|
The pre.4-to-HEAD storage blocker described above is now addressed by commit The existing raw-wire compatibility layer now traverses message-valued protobuf maps and converts legacy The remaining confirmation is the label-gated DEB/RPM qualification run using packages built from this commit. |
|
/ok-to-test 262732e |
BlockedGator is blocked because this pull request is still a draft. Next action: @elezar, mark the pull request ready for review when the upgrade qualification work is ready for code review. Gator will review the current head in the next watch cycle after that transition. Gator metadata
|
cbf12c6 to
ee1528d
Compare
|
/ok-to-test ee1528d |
BlockedGator is blocked because this pull request is still a draft and the current head has merge conflicts with Next action: @elezar, resolve the conflicts with Gator metadata
|
348a5c3 to
5601c2c
Compare
86a5838 to
236eab4
Compare
fbdb8c1 to
febf3be
Compare
e51715c to
fe82250
Compare
295e946 to
f2f756f
Compare
|
Label |
SDAChess
left a comment
There was a problem hiding this comment.
My agent found two issues with keeping the continuity lane advisory. Both can affect the release flow even though continuity is excluded from the qualification result.
f580d45 to
d3f14ae
Compare
…ption Add a `continuity` e2e family whose scenarios are `seed` and `verify` test pairs sharing state through a run-scoped id. With `--disruption <name>` the test runner seeds, applies the named disruption in the guest, waits for the gateway, and then verifies, so scenarios stay independent of the disruption. The runner keeps a registry of disruption names. How a disruption is carried out depends on how OpenShell is installed, so the installer provides it as an executable under /var/lib/openshell-qualification/disruptions, and the runner checks that it exists before it seeds anything. The packaged-gateway role renders gateway-restart for the user that runs the gateway. The e2e matrix gains a `disruption` column and a `continuity/gateway-restart` row on ubuntu-docker-rootful with the deb installer. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Add an `upgrade` disruption that installs a candidate Debian package over a released baseline, switches the gateway to the candidate's runtime images, restarts it, and fails if the installed package is not the candidate. The scenarios stay unchanged and run under it like any other disruption. The `deb-upgrade-source` installer provisions the guest from a released package and its release-tagged images, and stages the candidate package and images under /var/lib/openshell-qualification/candidate together with the `upgrade` disruption that installs them. The packaged-gateway role keeps the release's own image configuration for a released source. Signed-off-by: Evan Lezar <elezar@nvidia.com>
Add an `upgrade` row to the continuity matrix that installs the candidate over the latest published release. The baseline is the release's Debian package and runtime images, staged under artifacts/upgrade by prepare-integration-inputs when a caller passes `include-deb-upgrade-source`; callers select the row with `upgrade-source`, as external drivers are. Add an optional `test:e2e:continuity` label to Branch E2E Checks that runs the continuity family on top of `test:e2e`, and run it in Release Dev and Pre-release qualification. It is not part of either release result yet, so a failure shows in the run without blocking the release. Document the label, the family and how to run it locally. Signed-off-by: Evan Lezar <elezar@nvidia.com>
d3f14ae to
9a30af5
Compare
|
Design direction for continuity, separate from the approval below. I'd like us to consider it before more scenarios are written against the current shape. seed/verify exists mostly because testkit cleanup is per-test. That forces the CLI-create + Proposal: add a Not asking for this in this PR. The cost grows with each scenario, so I'd like a follow-up issue before we add more. I haven't checked whether the SDK channel recovers across a gateway restart; that's worth confirming in a prototype. |
matthewgrossman
left a comment
There was a problem hiding this comment.
Approving as a first cut so the deb and rpm work can land. The structure is good: scenarios stay disruption-agnostic and the installer supplies the how. The continuity rows pass in CI. Design direction is in my separate comment.
Small notes, none blocking:
- A gateway that doesn't come back after a disruption is reported as a runner error. For an upgrade test that is the primary failure, so I'd report it as a test failure.
upgrade_source_tagis the latest release at run time, so the same candidate can pass one day and fail the next. Consider pinning it, or recording the tag in the summary.- Neither this nor #4202 exercises
install.sh, which is the path that broke. I'll addinstall.shinstallers on this disruption contract under #5740, so no change needed here.
Yes, that should probably change. It may be easier to manage / detect with the testkit-based disruption that we'll look into as a follow-up.
I don't think we should pin it. That means that we'd have to keep it up to date as part of the release process to ensure that we're testing N-1 -> N. I agree that we could record the tag in the summary.
Yes, I am aware of this, but we need |
Summary
Add a
continuitye2e family that checks state created before a disruption survives it, with two disruptions: a gateway restart and a package upgrade from the latest published release. Scenarios areseed/verifytest pairs that run unchanged under every disruption, so a new disruption such as a rollback or a reboot needs no new test design.RPM upgrade coverage stays in the stacked follow-up #4202.
Related Issue
Refs #2975 — reusable N-to-N+1 upgrade qualification.
Changes
continuitytestkit family (tests/suites/continuity). A scenario'sseedcreates state through the CLI, so testkit cleanup leaves it in place, and itsverifyadopts that state, checks it, and creates something new. The first scenario writes a marker into a sandbox's workspace. Both tests derive names from a run id (CONTINUITY_RUN_ID).openshell-test-runner run --disruption <name>, which runs theseedtests, applies the disruption in the guest, waits for the gateway, and runs theverifytests./var/lib/openshell-qualification/disruptions/, because how to restart or upgrade OpenShell depends on how it is installed. The runner keeps only the registry of names and checks that the installer provides the one requested before it seeds anything.gateway-restart, rendered by the packaged-gateway role for the user that runs the gateway.upgradeand thedeb-upgrade-sourceinstaller. The installer provisions the guest from the latest release's package and its release-tagged images, and stages the candidate. The upgrade step installs the candidate, requires the installed version to equal the candidate's, switches the gateway to the candidate's runtime images, and restarts it.upgraderow itself, with theprepare-upgrade-sourceaction and a release tag thatselectresolves once, so a failed download affects only that row and not the sharedprepare-integrationjob. Select the row withupgrade-source, as external drivers are selected.test:e2e:continuitylabel that runs the family on top oftest:e2e, and run it in Release Dev and Pre-release. Release Dev and the qualification result need the continuity jobs, like conformance and feature integration, and the qualification summary records them.TESTING.mdandCI.md.Release Notes
Testing
Ran on macOS with HVF on
ubuntu-docker-rootful, with the baseline v0.1.9 and the candidate built fromed8a0d98a:continuitywithgateway-restarton thedebinstaller: passes.continuitywithupgradeondeb-upgrade-source: passes. The same run failed twice before for the right reasons: a stale candidate whose database migrations were older than the baseline's, and a wrongly tagged candidate image.upgradeagainst thedebinstaller, which doesn't provide it: stops before seeding with an error naming the missing executable.openshell-test-runner:clippy -D warnings,rustfmt --checkand unit tests pass.test-qualification-summary.shpasses, and an earlier head passedactionlint, with the row selection run withupgrade-sourceon and off. Existing row and artifact names are unchanged.markdownlintandnix build .#tmachine-configpass.Not verified:
The new CI jobs have not completed yet, including the baseline download and image export, and
actionlintwas not rerun on this head.The pre-commit hooks were skipped:
rust:lintcrashes with a clippy internal compiler error inopenshell-supervisor-network, which this PR doesn't touch.A gateway that fails to come back after a disruption is reported as a runner error, not a test failure.
Checks appropriate to the affected code and behavior pass
Unit tests added/updated (if applicable)
E2E tests added/updated (if applicable)
Checklist